ZA CHOICE 


Working Paper 2020-4 


ONLINE LEARNING: HOW DO BRICK AND MORTAR 
SCHOOLS STACK UP TO VIRTUAL SCHOOLS? 


lan Kingsbury 
iank@empirecenter.org 


September 30, 2020 


ABOUT EDCHOICE 


EdChoice is a nonprofit, nonpartisan organization dedicated to advancing full and unencumbered 
educational choice as the best pathway to successful lives and a stronger society. EdChoice believes that 
families, not bureaucrats, are best equipped to make K—12 schooling decisions for their children. The 
organization works at the state level to educate diverse audiences, train advocates and engage 
policymakers on the benefits of high-quality school choice programs. EdChoice is the intellectual legacy 
of Milton and Rose D. Friedman, who founded the organization in 1996 as the Friedman Foundation for 
Educational Choice. 


The contents of this publication are intended to provide empirical information and should not be 
construed as lobbying for any position related to any legislation. 


EdChoice Working Papers are meant for wide circulation, inviting reader feedback, and for informing 
national and state conversations about reform ideas, policymaking, and implementation, driven by 
educational choice. EdChoice internally reviews each paper but does not necessarily send out for external 
reviews. 


We welcome and encourage hearing from you. Please feel free to contact the author directly — or Paul 
DiPerna, our Vice President of Research and Innovation — about your impressions and comments. 


Ian Kingsbury: iank@empirecenter.org 


Paul DiPerna: paul@edchoice.org 


Authors are responsible for the content provided in EdChoice working papers. 


Table of Contents 


Executive Summary 

Introduction 

Methods 

Results and Discussion 
Active Learning 
Communication 
Classroom Management 
Pedagogical Efficacy 
Subgroup Analysis 

Conclusion 

References 

About the Author(s) 


Acknowledgments 


List of Tables 


Table 1: Virtual Learning Constructs 

Table 2: Communications Index Statistics 

Table 3: Classroom Management Index Statistics 
Table 4: Pedagogical Efficacy Index Statistics 
Table 5: Scores by Racial Group 

Table 6: Scores by School Sector 


Executive Summary 


The Covid-19 pandemic forced many American schools to hastily transition to online learning. I assess 
how the online learning experience of students enrolled in brick and mortar schools that transitioned to 
online learning in Spring 2020 compared to the experience of students who were already enrolled in 
virtual schools when the pandemic began. Absent formal assessments to quantify learning loss, such 
comparison can help contextualize the performance of brick and mortar schools in their transition to 
online learning. Moreover, comparison informs how policy can promote higher-quality online schooling, 
a burgeoning policy concern amidst widespread school closures forecasted for the 2020-2021 academic 
year. 


I hypothesize that, owing to experience and expertise, virtual schools provided a higher quality education 
than did brick and mortar schools operating online. I test this hypothesis by administering surveys to 
parents of students enrolled in virtual schools. When applicable, parents also complete a survey about the 
online learning experience of siblings enrolled in brick and mortar schools that switched to online 
learning in Spring 2020. I compare survey outcomes across four constructs which research indicates are 
critical to successful online learning: active learning, communication, pedagogical efficacy, and 
classroom management. 


Overall, I observe that virtual schools earned substantially higher marks across the four constructs. 
Parents were significantly more likely to report that their child engaged in activities associated with active 
learning, and they agreed that virtual schools outperformed brick and mortar schools when it comes to 
clear communication, classroom management, and sound instructional practices. On the three indexes for 
which I derive a composite index score-communication, classroom management, and sound instructional 
practices-virtual schools outperformed brick and mortar schools by more than one standard deviation. 


One question within the pedagogical efficacy construct simply asks whether parents feel that their 
children learned a lot. The responses were more divergent according to school sector than any 
agree/disagree question in the survey. Whereas 86.4% of respondents agreed or strongly agreed that their 
children in virtual schools “learned a lot,” only 13.4% of respondents of brick and mortar students agreed 
or strongly agreed that their children learned a lot. The radically divergent responses to that question 
highlight that learning is, to some extent, a function of all the items on this survey. Students in virtual 
schools plausibly learned more because they benefited from higher quality instruction, better classroom 
management, clearer communication, and more frequent active learning. 


I examine differences in responses by racial groups and school sector (i.e. traditional public, charter 
school, or private school) to assess concerns that school closures could exacerbate socioeconomic 
achievement gaps. Overall, I observe mixed evidence. Private schools, which are cost-prohibitive for 
many families, earn higher marks than public schools. On the other hand, Hispanic and African American 
parents report a better online learning experience in brick and mortar schools than do white or Asian 
parents. Moreover, charter schools, which disproportionately serve students of color and underprivileged 
students, earned higher marks than traditional public schools for their handling of online learning. 


When it comes to drawing policy conclusions, some limitations apply. First, data was collected from 
parents of students enrolled in schools affiliated with K12 Inc. Though K12 Inc. is the largest provider of 
online primary and secondary education, it is unclear to what extent schools affiliated with K12 Inc. are 
representative of the virtual schooling sector. In other words, it is not clear to what extent survey results 
reflect differences between brick and mortar schools and K12 Inc.-affiliated schools or between brick and 
mortar schools and the universe of virtual schools. Second, it remains to be seen to what extent brick and 


mortar schools might improve their virtual instruction during the 2020-2021 academic year. Depending 
on when schools or districts announced their reopening plans, teachers would have only had weeks or 
months to prepare for online instruction. Still, weeks or months of preparation is an improvement over the 
hasty closings which occurred in Spring 2020. Finally, while one expects that survey results are correlated 
with student achievement—the outcome that conventionally guides education policymaking—the strength 
of that correlation is unclear. 


Caveats notwithstanding, the magnitude of differences in survey responses almost certainly indicate that 
students in virtual schools learned more than students in brick and mortar schools operating virtually. The 
magnitude of difference in learning may shrink, though is unlikely to disappear, during the current 
academic year. Consequently, in the interest of student learning, states should not cap, and should perhaps 
seek to expand access to virtual schools, at least through the course of pandemic-related school closures. 


Introduction 


Thousands of American schools shuttered their doors during the second half of the 2019-2020 school year 
due to the Covid-19 pandemic. Most schools that closed their physical campus switched to online 
learning to conclude the school year. Brick and mortar schools uninitiated to online learning were tasked 
with adapting teaching practices to a different modality with almost no training or experience to call 
upon. While anecdotal accounts indicate that some schools achieved remarkable success in transitioning 
to distance modalities of learning, many others schools struggled to keep students and teachers engaged in 
the learning process. 


There is great urgency in assessing the overall efficacy with which brick and mortar schools executed 
online learning. Many American schools are beginning the 2020-2021 academic year with virtual or 
blended models of schooling, while countless others scheduled to open will inevitably transition to virtual 
or blended learning as public health concerns mount. 


Difficulties in transitioning to online learning elevate concerns about national competitiveness. A 
comparatively greater number of schools in other developed regions of the globe (e.g. Europe and East 
Asia) are poised to begin the 2020-2021 school year in person, and are perhaps likelier to sustain in- 
person activities due to a lower incidence of Covid-19 among the general population. If American brick 
and mortar schools struggle in their delivery of online education, the human capital stock of Generation Z 
and Generation Alpha could languish compared to other developed nations (Psacharopoulos et al., 2020). 


Difficulties in transitioning to online learning also elevate concerns about equity. Privileged families 
unsatisfied with the virtual learning experience delivered by their public schools paid for alternatives, 
including private schools, microschools, or learning pods. Others homeschooled their children, an option 
with limited direct cost but potentially high opportunity cost (i.e. loss of parental income). 


For some families who pursued alternatives to their residentially assigned public school, the chief impetus 
was a concern that virtual education cannot replicate in-person schooling in terms of academic 
effectiveness or that it could not adequately meet the social-emotional needs of their children. For others, 
perhaps, the concern was not the modality of schooling itself but the quality of instruction offered by the 
school, as evidenced by strong demand for established virtual schools (Tanner, 2020; Associated Press, 
2020). 


Though some scholars question the wisdom and limitations of online education, expanded utilization 
appears to be a foregone conclusion. In the short run, millions of families will utilize virtual education 


due to safety concerns surrounding Covid-19 or resource constraints which compel them to enroll in a 
school operating virtually, even if it is not their preference. In the long run, it is likely that a non- 
negligible number of families compelled to participate in virtual schooling will discover that it better 
meets the needs of their child (Schroeder, 2020), and they will enroll in virtual schools regardless of the 
broader public health situation. Indeed, a survey conducted in May 2020 found that 73% of parents would 
be willing to have their children take at least some high school courses online, a 17-percentage point 
increase from 2009 (Henderson et al., 2020). 


Methods 


The recency of Covid-19 and dearth of formal assessments administered since schools closed renders 
impossible a causal evaluation of learning loss incurred by students enrolled in brick and mortar schools 
that switched to online learning during the 2019-2020 school year. Nevertheless, given the volume of 
students who will experience at least some online learning during the 2020-2021 school year, there is an 
urgent demand to understand as best as possible what went well and what did not go well after brick and 
mortar schools switched to online learning. 


I offer preliminary evaluation of how schools handled the transition to online learning by comparing the 
online learning experience of students enrolled in brick and mortar schools that switched to online 
learning during the 2019-2020 school year to students who were enrolled in virtual schools before the 
pandemic struck. While online schools have faced scrutiny for their performance, their years of 
institutional knowledge, expertise, and virtual learning infrastructure should in theory make them more 
adept than brick and mortar schools at delivering virtual instruction. So while this study does not weigh 
in—philosophically or empirically—about the overall effectiveness of online schooling compared to in- 
person schooling, it offers a first glimpse at how well brick and mortar schools handled virtual learning 
compared to established virtual schools. 


Two anonymous surveys were administered to parents of children enrolled in school powered by K12, a 
provider of online schooling and curricula. One survey asks parents to reflect on the experience of their 
child enrolled in a virtual school; the other asks them whether they have children who were enrolled in 
brick and mortar schools during the 2019-2020 academic school year and, when applicable, to reflect on 
their experience with virtual learning through that brick and mortar school. 99,826 surveys were 
deployed, of which 10,144 were answered (10.1% response rate). 61% of submitted surveys were 
completed on behalf of students enrolled in virtual schools. The remainder were completed on behalf of 
students enrolled in brick and mortar schools that switched to online instruction. Two thirds of 
respondents identified their child as white whereas 13% identified themselves as African American, 13% 
as Hispanic or Latino, and 2% as Asian. The demographic composition of the sample mirrors the national 
demographic composition of virtual school students (Molnar, 2019). 


Ordinal questions prompt respondents to assess the degree to which they agree with a statement (1, 
Strongly disagree; 2, Disagree; 3, Somewhat disagree; 4, Neither agree nor disagree; 5, Somewhat agree; 
6, Agree; 7, Strongly agree) or the frequency with which an event occurred (1, Never; 2, Once per month 
or less; 3, A couple times per month; 4, About once per week; 5, More than once per week). Respondents 
were also given the option of responding that they were uncertain of their response to a question. 
Uncertain responses are coded as missing data. Questions are assigned to one of four constructs. Those 
constructs, the questions that comprise them, and Cronbach’s alpha coefficients are displayed in Table 
One. Overall, the constructs display a high level of internal consistency. 


Table One: Virtual Learning Constructs 


Active Learning (Response Scale: Frequency) 


Item Alpha 
Instructors assigned group projects and/or 84 
team-based tasks 

Instructors required students to participate in 84 
discussion forums (either written or spoken) 

Instructors required students to share their 79 
work with other students to receive feedback 

Instructors provided opportunities for students .82 

to present their work to the entire class 
Test Scale .86 


Communication (Response Scale: Agreement) 


Item Alpha 
Instructors set clear academic standards 94 
Instructors provided timely feedback to students .90 
on completed work (e.g. papers or tests) 
Instructors made themselves available to help .90 
my child/children outside of classroom hours 
Instructors answered my calls, texts, or emails in .90 
a timely fashion 
Accessing course materials was easy and 91 
straightforward 
Test Scale 91 


Pedagogical Efficacy (Response Scale: Agreement) 


Item Alpha 

Instructors motivated students to do their best .96 

Instructional materials worked well for learning 95 
in an online/virtual setting 

| feel like my child learned a lot 95 

Instructors motivated students to care about 95 

what they were learning 

Instructors taught new material rather than .96 
simply review old material 

Test Scale .96 


Classroom Management (Response Scale: Agreement) 


Item Alpha 


Instructors were skilled at preventing and 94 
managing disruptions in the virtual classroom 


The pace of learning was well-suited to the 94 
needs of my child 
Attendance and participation were expected 95 
and tracked 
There was a clear learning plan for each week 94 
Instructors demonstrated competency with 94 
operating virtual classroom software 

Test Scale 95 


Results and Discussion 


I hypothesize that virtual schools would outperform brick and mortar schools in their delivery of online 
education. The specific purpose of this study is to glean the magnitude of that advantage and assess 
relative strengths and weaknesses by surveying parents to assess efficacy within four constructs that 
research indicates are critical to successful virtual education. Overall, the survey results indicate that 
virtual schools significantly outperformed brick and mortar schools on each index and all items which 
comprise the index. Differences in outcomes on all items are statistically significant at the 99% 
confidence level. 


Active Learning 


Active learning entails encountering new information or ideas, engaging with them, and reflecting on 
what was learned (Fink, 2003). A meta-analysis of studies conducted on active learning indicates that it is 
associated with stronger learning outcomes compared to transmissionist approaches in which teachers 
impart knowledge which students are tasked with absorbing (Freeman et al., 2014). 


Active learning is especially important for successful distance learning. Nummenmaa & Nummenmaa 
(2008) observe a link between evaluations of online courses and the degree of interactivity demanded by 
the course. That is, students have stronger appraisals of online courses which, per Moore’s (1989) 
typology, leverage learner-content interaction, learner-instructor interaction, and learner-learner 
interaction. 


Interaction, the process of reciprocal communication or exchange, improves learning through two non- 
mutually exclusive channels. First, transactional distance theory holds that the sense of obligation that 
learner and teacher feel toward one another is closely linked to the frequency and duration of their 
interactions. To that end, interaction can make the learner more invested in their own learning, thereby 
increasing effort. Second, the social-constructivist view of learning posits that interaction with peers, 
especially those of high intelligence and diverse backgrounds, is foundational to cognitive development. 


Preliminary evidence suggests that active learning—which is achieved largely through the three types of 
interactions in Moore’s typology—was deficient in many schools that switched to virtual instruction 
during the 2019-2020 school year. For example, media accounts indicate that some teachers simply 
assigned work packets for students to complete but were otherwise largely unengaged in the learning 
process (Kamenetz, 2020). The parents polled for this survey corroborate the notion that brick and mortar 
schools mostly neglected the active learning process. Indeed, certain results are both striking not only in 


their comparison to virtual schools but in absolute terms. For example, 69.4% of parents reflecting on the 
experience of their children in brick and mortar schools report that their children were never tasked with 
participating in group projects or team-based tasks. A similar percentage of parents of virtual school 
students reported that their children participated in group projects or team-based tasks at least a couple 
times per month. 


Participate in group projects and/or team-based 
tasks 


Brick and Mortar Schools 


Virtual Schools 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Virtual Schools Brick and Mortar Schools 
m@ More than once per week 1044 170 
m@ About once per week 798 285 
mA couple times per month 737 139 
m@ Once per month or less 633 241 
m Never 575 1893 


Participate in discussion forums 


Brick and Mortar Schools 


Virtual Schools 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Virtual Schools Brick and Mortar Schools 
@ More than once per week 2,378 480 
m@ About once per week 820 594 
mA couple times per month 579 223 
m Once per month or less 307 273 


@ Never 185 1165 


Share work with other students to receive feedback 


Brick and Mortar Schools 


Virtual Schools 


| 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Virtual Schools Brick and Mortar Schools 
m@ More than once per week 1015 204 
m@ About once per week 768 241 
mA couple times per month 764 187 
m Once per month or less 582 231 
m Never 548 1810 


Present work to the entire class 


Brick and Mortar Schools 


Virtual Schools 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Virtual Schools Brick and Mortar Schools 
@ More than once per week 1145 165 
m About once per week 700 299 
mA couple times per month 796 218 
m@ Once per month or less 755 421 
m Never 356 1583 


Overall, 39.5% of respondents with children enrolled in brick and mortar schools that switched to online 
learning report that their children never participated in any of the activities which comprise the active 
learning construct, whereas 2.1% of virtual school parents report that their children never participated in 
such activities. Meanwhile, 46.1% of respondents reported that their children in brick and mortar schools 
did not participate in any of the four activities more than a couple times per month, whereas 4.5% of 
respondents make the same claim about children enrolled in virtual schools 


Note that I do not tabulate a composite index score for this construct, as the responses offered to the 
questions in the active learning construct are not quantitatively equal in their magnitude of difference 
from one another. I elected for precisely defined frequency responses rather than Likert-scale responses 
because Likert-scale responses are vague and difficult to interpret when they address event frequency. 


Communication 


Clear communication from teachers to students and parents can pose challenges in a virtual format, which 
demands different skills and practices from clear communication in a traditional face-to-face format. 
Ensuring that students are clear on learning expectations presents a challenge for teachers accustomed to 
meeting with students in person several days per week (Isman et al., 2003). Moreover, the challenge of 
establishing and maintaining clear communication between teachers and families might be exacerbated by 
anxiety and concomitant loss of focus experienced by many through the course of the Covid-19 pandemic 
(Canle, 2020). Challenges notwithstanding, student perceptions of teacher communication are critical to 
student learning within distance schooling modalities, and unclear expectations are specifically cited as a 
common source of frustration (So & Brush, 2008). 


Quality of communication from instructors to students and parents is critical to successful online learning. 
Teachers must be clear in their expectations for student work and conduct during synchronous sessions 
and “must establish the purpose of assessment, the criteria being measured, and the intended outcomes 
before meaningful assessment methods can be achieved.” (Gaytan & McEwen, 2007, p. 118). Quantity of 
communication matters, too. Teachers must respond to student inquiries in a timely fashion to overcome 
transactional distance and practical limitations imposed by the absence of face-to-face interaction. For 
that reason, some virtual schools require that teachers respond to all e-mails and phone calls within 24 
hours (Watson et al., 2009). 


The virtual learning construct asks whether instructors answered phone, text, or email inquiries in a 
timely fashion, and whether instructors were accessible outside of classroom hours. It also gauges 
communication across key stages in the teaching process, including whether instructors set clear academic 
standards, whether course materials were easy to access, and whether they provided timely feedback on 
student work. 


Instructors set clear academic standards 


Brick and Mortar Schools 


Virtual Schools 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Virtual Schools Brick and Mortar Schools 
@ Strongly agree 2568 239 
m Agree 1756 419 
m& Somewhat agree 380 550 
m@ Neither agree nor disagree 126 276 
@ Somewhat disagree 75 339 
@ Disagree 31 618 


@ Strongly Disagree 25 493 


Instructors provided timely feedback on completed 
work 


Brick and Mortar Schools lll es 
Virtual Schoo!s Sr 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Virtual Schools Brick and Mortar Schools 
@ Strongly agree 2363 295 
m Agree 1668 594 
m& Somewhat agree 485 482 
g Neither agree nor disagree 133 252 
m Somewhat disagree 104 313 
m@ Disagree De} 504 
@ Strongly Disagree 30 464 


Instructors made themselves available to help 
outside classroom hours 


Brick and Mortar Schoo! lll 
Virtual Schools |S 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Virtual Schools Brick and Mortar Schools 
@ Strongly agree 2609 386 
m Agree 1452 561 
m Somewhat agree 369 572 
g Neither agree nor disagree 229 316 
m Somewhat disagree 87 319 
m@ Disagree 58 429 


@ Strongly Disagree 36 332 


Instructors answered calls, texts, or emails in a 
timely fashion 


Brick and Mortar Schools SSN i pers 
Virtual Schools SS rm 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Virtual Schools Brick and Mortar Schools 
@ Strongly agree 2724 560 
@ Agree 1438 831 
m Somewhat agree 374 561 
@ Neither agree nor disagree 137 296 
m Somewhat disagree 73 252 
m@ Disagree itl 215 
@ Strongly Disagree 26 193 


Accessing course materials was easy and 
straightforward 


Brick and Mortar Schoo — lll 
Virtual Schools | 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Virtual Schools Brick and Mortar Schools 
m Strongly agree 2509 256 
m Agree 1589 570 
& Somewhat agree 438 564 
W@ Neither agree nor disagree 117 316 
m Somewhat disagree 92 356 
@ Disagree 30 386 
m Strongly Disagree 32 443 


TABLE TWO: COMMUNICATIONS INDEX STATISTICS 


Virtual Schools Brick and Mortar Schools 
25™ PERCENTILE 6 3 
50™ PERCENTILE 6.40 4.20 
75™ PERCENTILE 7 5.40 
MEAN 6.30 4.14 
STANDARD DEVIATION 0.87 1.55 


All told, parents evaluating the clarity and frequency of communications from instructors to students and 
families give considerably higher marks to virtual school staff. The average communications score for 
virtual schools is 6.3, and the median is 6.4, indicating that parents typically fall in the range of either 
agreeing or strongly agreeing that communication was clear and frequent. Parents of brick and mortar 
students who received virtual instruction receive an average score of 4.1 and a median score of 4.2, scores 
that fall closer to neutrality than modest agreement. Unsurprisingly, there was also greater variation in 
how the parents of brick and mortar students evaluated communications. Whereas the virtual school 
communications index score has a standard deviation of .87 points, the brick and mortar index score has a 
standard deviation of 1.55 points. Consequently, virtual schools in the 25" percentile receive an overall 
score of 6, indicating agreement with clear and frequent communication. Meanwhile, brick and mortar 
schools in the 25" percentile earn a score of 3, indicating that parents “somewhat disagree” that 
communication was effective. 


Classroom Management 


In devising a conceptual model for virtual classroom management, Rufai, Alebiosu & Adeakin (2015) 
begin by stating, “The primary purpose of teaching is to impact knowledge. However, two factors can 
facilitate this objective. One is having a simplified and well explained course material and the other is 
ensuring its effective delivery. A well-managed classroom will guarantee effective delivery.” (p. 27). 
While this perhaps overstates the importance of classroom management—indeed, a teacher can employ 
pedagogically unsound practices in an otherwise well-managed classroom—it highlights that classroom 
management—“the act of supervising relationships, behaviors, and instructional settings and lessons for 
communities of learners” (Iverson, 2003, p. 4)—is a necessary (if insufficient) condition for student 
growth within virtual and in-face schooling modalities. 


The classroom management construct contains questions about the skill with which instructors prevented 
disruptions during synchronous instruction, as there were many accounts of students or unwanted visitors 
disrupting learning, or even sharing lewd content during Zoom sessions (Stevens, 2020). The construct 
also asks about pace of learning. Depending on design, distance learning can empower students to learn at 
their own pace, but to what degree this occurred in brick and mortar schools that closed down is unclear. 
Finally, the construct gauges whether attendance and participation were tracked, whether teachers had a 
clear learning plan for each week, and whether instructors demonstrated competency with virtual 
classroom software. 


Instructors were skilled at preventing and 
managing disruptions 


Brick and Mortar Schools lilly lla ees 
Virtual Schools Sammy 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Virtual Schools Brick and Mortar Schools 
@ Strongly agree 2351 160 
m Agree 1760 338 
m= Somewhat agree 317 286 
g@ Neither agree nor disagree 264 941 
m Somewhat disagree 65 254 
m@ Disagree 33 436 
@ Strongly Disagree 32 487 


The pace of learning was well-suited to the needs 
of my child 


Brick and Mortar Schoo lll laa rs 
Virtual Schools | 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Virtual Schools Brick and Mortar Schools 
@ Strongly agree 2297 173 
m Agree 1653 347 
m Somewhat agree 459 356 
g Neither agree nor disagree 147 421 
m Somewhat disagree 144 340 
@ Disagree 57 547 


@ Strongly Disagree 45 706 


Attendance and participation were expected and 
tracked 


Brick and Mortar Schools SS ees 


Virtual Schoo|s | er 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Virtual Schools Brick and Mortar Schools 
@ Strongly agree 3057 412 
@ Agree 1474 559 
m Somewhat agree 138 367 
@ Neither agree nor disagree 89 265 
m Somewhat disagree 24 250 
m@ Disagree 11 452 
@ Strongly Disagree 16 595 


There was a clear learning plan for each week 


Brick and Mortar Schools 


Virtual Schools 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Virtual Schools Brick and Mortar Schools 
@ Strongly agree 2763 300 
m Agree 1566 579 
mj Somewhat agree 290 549 
g Neither agree nor disagree 99 272 
m Somewhat disagree 54 331 
@ Disagree 17 364 


@ Strongly Disagree 25 502 


Instructors demonstrated competency with 
operating virtual classroom software 


Brick and Mortar School jill ia es 
Virtual Schools Sa rrr 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Virtual Schools Brick and Mortar Schools 
@ Strongly agree 2514 166 
m Agree 1705 371 
& Somewhat agree 338 415 
g Neither agree nor disagree 165 476 
m Somewhat disagree 51 353 
m@ Disagree 23 497 
@ Strongly Disagree 20 619 


TABLE THREE: CLASSROOM MANAGEMENT INDEX STATISTICS 


Virtual Schools Brick and Mortar Schools 
25™ PERCENTILE 6 2.40 
50" PERCENTILE 6.40 3.60 
75" PERCENTILE 7 4.80 
MEAN 6.32 3.66 
STANDARD DEVIATION 0.79 1.56 


Parents give strong marks to virtual school teachers for their managing of a virtual classroom. On 
average, parents agree to strongly agree that classrooms were skillfully managed, and three quarters either 
agree or strongly agree with that premise. Parents are less bullish on how classroom management was 
handled in brick and mortar schools. On average, parents express neutrality to modest disagreement with 
the notion that online classrooms were well-managed. Unsurprisingly, while the superior assessments of 
classroom management in virtual schools is significant on all items within the construct, it is primarily 
driven by perceived competency with managing virtual classroom software, a phenomenon likely 
explained by pronounced asymmetries in software experience and expertise between virtual school 
teachers and brick and mortar teachers. 


Pedagogical Efficacy 


Ultimately, virtual schools are judged according to the same criterion upon which other schools are 
judged: Chiefly, to what extent did the school produce knowledge and skill acquisition? A 2004 meta- 
analysis analyzing the effects of web-delivered K-12 programs vis-a-vis academic outcomes concludes 
that, on average, virtual schools are indistinguishable from brick and mortar schools in the degree to 
which they produce gains in student achievement (Cavanaugh et al., 2004). A more recent Department of 


Education-sponsored meta-analysis authored by Means et al. (2009) concludes that online learning is 
modestly more effective than in-person learning, but the study cautions that there are a limited number of 
studies from K-12 education, and only five qualify as experimental or quasi-experimental. Moreover, “In 
many of the studies showing an advantage for online learning, the online and classroom conditions 
differed in terms of time spent, curriculum and pedagogy...the studies in this meta-analysis do not 
demonstrate that online learning is superior as a medium.” (Means et al., 200, XVII). 


Both meta-analyses illustrate significant variation in effect sizes among different programs, so meta- 
analytic averages can obfuscate the fact that virtual school programs sometimes perform significantly 
better and sometimes significantly worse than in-person programs. Miron & Urschel (2012) attempt to 
gauge where K12 Inc.-operated schools fall within this distribution of outcomes and conclude that 
performance indicators indicate weak performance. However, Chingos (2013) criticizes their conclusions 
because they rely upon achievement data rather than growth data. Such data is more instructive about the 
characteristics and demographics of students enrolled in a school than it is about the quality of the school 
itself. Unfortunately, then, comparison between K12 Inc.-affiliated virtual schools and brick and mortar 
schools cannot directly inform the degree to which student learning in the latter was impacted by the 
switch to virtual learning. Again, I emphasize that the purpose of this endeavor is to compare student 
experiences between the virtual schooling sector and the brick and mortar sector operating virtually. 


To date, no assessment data has been collected subsequent to campus closures. Right now, then, the most 
instructive data comes from assessments and comparisons across sectors (brick and mortar versus virtual) 
of pedagogical efficacy. That is, to what extent did teachers succeed in promoting student cognitive 
development and knowledge acquisition? The learning efficacy construct assesses this question first and 
foremost by asking parents whether they feel their children learned a lot. It also inquires whether teachers 
covered new material—state guidance varied widely on this matter (Schwartz, 2020)—-whether teachers 
motivated students to care about their learning (Theobald, 2006), and whether instructional materials were 
well-suited for virtual learning. 


Instructors motivated students to do their best 


Brick and Mortar Schools 


Virtual Schools 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Virtual Schools Brick and Mortar Schools 
m@ Strongly agree 2830 428 
@ Agree 1466 627 
m& Somewhat agree 301 521 
@ Neither agree nor disagree 142 383 
@ Somewhat disagree 44 282 
@ Disagree 29 356 


@ Strongly Disagree 27 311 


Instructional materials worked well for learning in 
an online/virtual setting 


Brick and Mortar School lll lars 
Virtual Schoo|s a errmmy 
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Virtual Schools Brick and Mortar Schools 
@ Strongly agree 2491 192 
m Agree 1696 324 
m Somewhat agree 382 441 
@ Neither agree nor disagree 134 329 
m Somewhat disagree 77 372 
m@ Disagree 29 602 
@ Strongly Disagree 27 644 


| feel like my child learned a lot 
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Virtual Schools Brick and Mortar Schools 
@ Strongly agree 2498 le nl 
m Agree 1660 257 
m Somewhat agree 383 365 
g Neither agree nor disagree 136 292 
m Somewhat disagree 72 350 
@ Disagree 30 564 


@ Strongly Disagree 32 944 


Instructors motivated students to care about what 
they were learning 


Brick and Mortar Schools lll la ees 
Virtual Schoo!s |S remy 
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Virtual Schools Brick and Mortar Schools 
@ Strongly agree 2342 204 
m Agree 1670 355 
m Somewhat agree 439 447 
@ Neither agree nor disagree 242 445 
mw Somewhat disagree 72 361 
m Disagree 44 520 
m Strongly disagree 26 571. 


Instructors taught new material rather than simply 
review old material 


Brick and Mortar Schoo _ lll la rs 
Virtual Schools | 
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Virtual Schools Brick and Mortar Schools 
@ Strongly agree 2153 185 
m Agree 1815 413 
m& Somewhat agree 402 447 
g Neither agree nor disagree 315 419 
m Somewhat disagree 53 329 
@ Disagree 39 456 
@ Strongly Disagree 30 634 


TABLE FOUR: PEDAGOGICAL EFFICACY INDEX STATISTICS 


Virtual Schools Brick and Mortar Schools 
254 PERCENTILE 6 2.20 
50™ PERCENTILE 6.40 3.40 
75" PERCENTILE 7 4.80 
MEAN 6.30 3.55 
STANDARD DEVIATION 0.86 1.62 


Assessments of the pedagogical efficacy of virtual schools almost mirror the results for communication. 
That is, three-quarters of parents either agree or strongly agree that teachers employed instructional 


practices that were conducive to student learning. Assessments from brick and mortar parents, however, 
appear unremarkable. On average, parents report somewhere between neutrality and modest disagreement 
with the premise that instructional practices were conducive to learning. 


One item within the construct deserves particular attention. A question asks whether parents feel that their 
children learned a lot. The responses were more divergent according to school sector than any 
agree/disagree question in the survey. Whereas 86.4% of respondents agreed or strongly agreed that their 
children in virtual schools “learned a lot,” only 13.4% of respondents of brick and mortar students agreed 
or strongly agreed that their children learned a lot. That this question would feature the largest divergence 
is not surprising, as learning is, to some extent, a function of all the items on this survey. Plausibly, 
success in one component of schooling is linked to success in another, just as failure in one is linked to 
failure in another. One could imagine, for example, that ambivalence toward responding to questions in a 
timely fashion tamped down on student enthusiasm for learning. 


Subgroup Analysis 


Researchers and policymakers worry that school closures could exacerbate long-standing racial and 
socioeconomic achievement gaps. I probe this concern by comparing assessments of brick and mortar 
online schooling performance from parents according to their racial self-identification, and whether they 
completed a survey for a child enrolled in a private school. Private school enrollment is a rudimentary 
proxy for wealth, as enrollment costs can be prohibitive for many families. 


Overall, I observe mixed signals regarding equity concerns. Parents of African American and Hispanic 
students give appreciably higher marks for the performance of brick and mortar schools than do white or 
Asian parents. Indeed, African American parents give higher marks for communication, pedagogical 
efficacy, and classroom management. They also report more frequent active learning activities. 
Differences in construct scores by race regarding the performance of brick and mortar schools are 
statistically significant at the 99% confidence level. Most tellingly, while 21.6% of white or Asian parents 
at least somewhat agreed that their children learned a lot while their brick and mortar school operated 
online, 33.2% of parents of Hispanic children and 39% of parents of Black children at least somewhat 
agreed that their children learned a lot. 


Table Five: Scores by Racial Group 


Black or Hispanic White or Asian 
K12 Inc Brick and K12 Inc Brick and 
Mortar Mortar 
Communication Mean 6.33 4.38 6.23 4.06 
SD 0.79 1.61 0.90 1.51 
Pedagogical Efficacy Mean 6.31 3.92 6.23 3.41 
SD 0.79 1.69 0.87 1.56 
Classroom Management Mean 6.39 4.05 6.33 3.50 
SD 0.69 1.63 0.79 1.57 


Differences in scores by school sector offer mixed evidence as to whether school closures will exacerbate 
achievement gaps. Among students in the sample who attended a brick and mortar school, 2,410 attended 
a traditional public school, 234 a private school, and 173 a charter school. Survey responses indicate that 
private school students received an appreciably better online education than did public school students, 


whether traditional public or charter school. Because private schools are unaffordable to many families, 
their relatively superior performance during school closures elevates concerns about the exacerbation of 
achievement gaps. On the other hand, parents gave higher marks to charter schools than to traditional 
public schools. Charter schools serve a disproportionately high number of students of color and students 
who qualify for free or reduced-price lunch, so the superior performance of charter schools over 
traditional public schools arguably dampens concerns regarding a widened achievement gap (Rebarber & 
Zgainer, 2014). 


All differences across sectors are statistically significant at the 99% confidence level except for 
communication between traditional public and charter school and classroom management between charter 
school and private school. Those differences are statistically significant at the 95% confidence level. 


Table Six: Scores by School Sector 


Traditional Public | Charter School Private School 
School 

Communication Mean 4.04 4.30 4.82 
SD 1.53 1.75 1.40 
Pedagogical Mean 3.40 3.78 4.46 
Efficacy SD 1.56 1.75 1.57 
Classroom Mean 3.47 4.17 4.50 
Management SD 1.57 1.50 1.50 


Conclusion 


Survey results indicate that students’ experiences with virtual learning in Spring 2020 varied markedly 
according to whether they were enrolled in brick and mortar schools or virtual schools. That outcome 
does not qualify as a surprise: Virtual schools would be expected to outperform brick and mortar 
counterparts that were forced to adapt to virtual learning with limited warning. The magnitude of 
difference, however, is jarring: Respondents were almost 6.5 times more likely to report that their child 
“learned a lot” in the Spring if they were enrolled in a virtual school. Moreover, the advantage in the 
performance of virtual schools in delivering online education is not easily attributable to their 
performance in any one facet of schooling. Rather, survey results indicate that virtual schools 
dramatically outperformed brick and mortar schools when it comes to promoting active learning, 
communicating effectively, managing a classroom, and providing high-quality instruction. The magnitude 
of difference was less among students of color, whose parents reported a substantially better experience 
with online instruction in brick and mortar schools than did white or Asian parents. The magnitude of 
difference was also less among charter school students compared to traditional public school students, and 
private school students compared to all public school students. 


Policy-relevant questions remain about these findings. For one, to what extent do parental assessments of 
learning predict achievement outcomes? As mentioned, questionnaires currently represent the best hope 
for gauging student learning during the pandemic, but it isn’t clear to what extent such measures proxy for 
actual student learning. Looking ahead, it’s also unclear to what extent the performance of brick and 


mortar schools might improve in Fall 2020 with regard to virtual teaching. On one hand, that many and 
perhaps most districts did not announce their reopening plans until July or August capped the amount of 
time for which teachers and administrators could prepare for a virtual reopening. On the other hand, a few 
weeks of preparation for virtual learning is an improvement over the hasty closings that schools were 
forced to conduct in the Spring. Finally, it is unclear to what extent the experience of students enrolled in 
schools affiliated with K12 Inc. are representative of the virtual schooling sector. 


Caveats notwithstanding, the radically divergent results revealed in this study suggest that, in prioritizing 
student learning, states would be imprudent to cap virtual school enrollment. On the contrary, in the 
interest of student learning, states should seek to expand access to established virtual schools through the 
course of pandemic-related school closures. 
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